Stochastic Control with Graphical Models: the Influence View Approach

| Markov decision processes (MDPs) allow to represent a wide class of problems in medical decision making and control. The complexity of the algorithms used to search the best policy of a MDP is directly related with the dimensionality of the state space. A careful structuring of the state space is hence an important task in the MDP spe-ciication. Graphical models are particularly appealing to cope with this task. In this paper we will describe a novel graphical formalism for MDP knowledge acquisition called Innuence View (IV). An IV is a directed acyclic graph that depicts the probabilistic relationships between the problem state variables in a generic time transition; additional variables, called event variables, may be added, in order to describe the conditional independencies between state variables. By using the speciied conditional independence structure, an IV may allow a parsimonious speciication of a MDP. The authors have applied this methodology to the GVHD prophylaxis after Bone Marrow Transplantation.